Skip to content

fix(adapters): fix exception mapping, transactions, and settings consistency across adapters - #875

Merged
cofin merged 174 commits into
mainfrom
adapter-remediation
Nov 2, 2026
Merged

cofin merged 174 commits into
mainfrom
adapter-remediation

Conversation

@cofin

@cofin cofin commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Summary

Adapter-wide correctness and consistency fixes from a review of all 22 database adapters. This change encompasses the cross-adapter contract fixes, web-framework and driver settings alignment, SQLite, MySQL, SQL Server/Sybase/SQL Anywhere/Db2, Oracle, PostgreSQL/CockroachDB family fixes, columnar adapters (ADBC, Arrow ODBC, BigQuery, DuckDB, Spanner), and a repository-wide adapter import contract.

The changes fix crashes and silent failures in exception mapping and transaction handling, align store and settings naming so framework integrations resolve every adapter, and tighten hot paths in the compiled adapter modules without shrinking the mypyc surface.

Key Changes

Core and extensions

  • SQLProcessor no longer treats T-SQL OUTPUT ... INTO as returning client rows.
  • FlaskConfig and StarletteConfig (and so FastAPIConfig) declare enable_correlation_middleware, correlation_header, correlation_headers, and auto_trace_headers, which the extensions already read; a contract test checks that every key an extension reads is declared.
  • enable_correlation_middleware now defaults to True in the Flask, Starlette, FastAPI, and Sanic extensions, matching Litestar.

Settings

  • Removed the pgbouncer and pgbouncer_mode driver_features aliases (asyncpg, psycopg, psqlpy) and the asyncpg connection_config["pgbouncer"] key; pooler_mode="transaction" is the single switch.
  • Renamed use_pool to enable_pooling (mssql_python, pymssql) and replaced psycopg null_pool with enable_pooling=False.
  • Removed the empty SybaseADKConfig, SqlanywhereADKConfig, Db2LitestarConfig, and Db2EventsConfig TypedDicts.
  • Renamed arrow_odbc enable_driver_pooling to enable_pooling and BigQuery use_query_and_wait to enable_query_and_wait.
  • Removed the ADBC strict_type_coercion and arrow_extension_types driver feature aliases; use enable_strict_type_coercion and enable_arrow_extension_types.
  • Declared driver feature keys that adapters already read: events_backend on ArrowOdbcDriverFeatures and default_query_job_config on BigQueryDriverFeatures.

Adapter import contract

  • Importing sqlspec.adapters.<name> imports that adapter's own driver eagerly in _typing.py; a missing driver raises the driver's own ModuleNotFoundError instead of failing later on placeholder values. Every driver stays optional and any adapter works without the others installed.
  • Secondary optional packages (pyarrow, numpy, pgvector, rich, google-cloud-bigquery-storage) load on first use; importing the oracledb, duckdb, psycopg, and CockroachDB psycopg adapters no longer pulls them in. The central sqlspec.typing shim gains ArrowException.
  • Spanner drops placeholder transaction and error classes and lazy driver exports; asyncmy no longer fails with re.error when asyncmy is missing.
  • tests/unit/test_import_time.py enforces the contract for every registered adapter, and docs/contributing/creating-adapters.md documents it for new adapters. sqlanydb joins the extras dependency group.

Store naming

  • Renamed SQLiteStore to SqliteStore, ADBCStore to AdbcStore, DuckdbStore to DuckDBStore, the DuckDB ADK classes to DuckDBADKStore, DuckDBADKMemoryStore, DuckDBADKConfig, and DuckDBADKFTSOptions, and the single-mode Sybase and SQL Anywhere event stores to SybaseEventQueueStore and SqlanywhereEventQueueStore. The Litestar session-table migration derives the store name from the config name, so it now resolves every adapter; a guard test covers all adapters with a Litestar store.

MySQL family (aiomysql, asyncmy, mysqlconnector, pymysql)

  • Errno 1061 and 1091 map to OperationalError instead of being suppressed, which previously crashed statement execution with an AssertionError; begin, commit, and rollback always raise the mapped error with its cause.
  • Removed unused helpers left over from splitting mysql_common.py into each adapter's core.py.
  • aiomysql escapes % in LOAD DATA statements; cached statement execution takes the cursor fast path again; literal-percent escaping has a count fast path; generator closures removed from compiled helpers.
  • Litestar stores drop the index-before-table ordering bug in DROP DDL and use a MySQL/MariaDB-compatible upsert; ADK memory inserts are batched with executemany; LIKE wildcards are escaped.
  • asyncmy maps additional errnos, guards its optional driver import, offloads LOCAL INFILE payload writes to a worker thread, and delegates script execution to the base driver.
  • MySQL data dictionary combines the DDL setup scalar queries and filters privileges by object name.

SQLite family (sqlite, aiosqlite)

  • Corrected the NOT NULL (1299) and CHECK (275) extended constraint codes; trigger and commit-hook aborts were previously reported as constraint violations.
  • sqlite tracks connection re-entrancy so nested scopes no longer end the outer transaction (commit-on-exit is unchanged), defaults :memory: to a shared cache, rejects invalid transaction modes with ImproperConfigurationError, and only applies substring error matching when no error code is available.
  • aiosqlite retires connections whose rollback fails instead of returning them to the pool, isolates ambient exceptions on cache hits, and maps ProgrammingError and InterfaceError.
  • ADK memory stores persist and search float32 embeddings; event queries parameterize LIMIT; generated-column metadata is memoized and schema names default to main.
  • Hot-path helpers hoisted to module scope, cursor creation deferred, and dead wrappers and redundant overrides removed.

SQL Server, Sybase, SQL Anywhere, Db2

  • Sybase and SQL Anywhere error bases bind to the native driver errors; transaction failures raise mapped errors (TransactionError for SQL Anywhere) with their cause.
  • mssql_python detects implicit transactions and rolls back on session exit, maps ODBC SQLSTATE codes, and fetches OUTPUT ... INTO results only when the cursor has a description; pymssql maps FreeTDS error codes and keeps an identical mapper.
  • Session MERGE statements use HOLDLOCK; ADK memory inserts are batched; unconfigured vector search is rejected; SQL Server data dictionaries implement the missing metadata methods.
  • Db2 translates migration schema errors, expands SQLSTATE mappings, reports pool checkouts, and batches ADK memory existence checks.
  • Sybase migration tracking resolves unqualified tables against USER_NAME(); SQL Anywhere reports supports_transactional_ddl=False and bounds its connection pool.
  • Added SAP SQL Anywhere integration driver test suite with hermetic in-container client runtime extraction.

Oracle (oracledb)

  • Corrected exception handling, script parameter binding, and transaction lifecycles; index-expression catalog queries no longer raise ORA-00997.
  • ADK memory stores add and persist the embedding column; ADK, Litestar, and events stores were remediated. AQ dequeue stays at-most-once.
  • Version caching moved into the compiled core; Arrow batches stream lazily; hot-path allocations reduced.

PostgreSQL and CockroachDB (asyncpg, psycopg, psqlpy, cockroach_asyncpg, cockroach_psycopg)

  • CockroachDB follower reads apply only to read-only queries and support streaming; the user psycopg configure callback runs after internal setup.
  • Exception mapping, transaction lifecycle, and script execution fixes across the three PostgreSQL drivers; psqlpy evaluates auth_token_provider at pool creation so tokens rotate.
  • ADK, Litestar, and events stores aligned; CockroachDB ADK memory inserts are batched and shard settings validated.
  • Data dictionary catalog queries fixed; cache-hit fast paths and fewer hot-path allocations.

Columnar (adbc, arrow_odbc, bigquery, duckdb, spanner)

  • ADBC handles autocommit in transactions, status codes, statement caching, and Snowflake index metadata; Litestar and ADK memory stores were hardened.
  • Arrow ODBC isolates ambient exceptions, maps ArrowException, begins transactions for atomic writes on SQL Server, and reports metadata capabilities.
  • BigQuery maps RetryError and GoogleAPIError, handles empty iterable schemas, preserves exception causes, and declares json_deserializer.
  • DuckDB handles divergent stream types, maps native exceptions and conflicts, implements metadata capabilities, and cleans temporary profiles.
  • Spanner guards transaction ownership, checks batch status codes, uses single-statement upserts and partitioned DML cleanup, maps DDL errors, keeps per-execute options concurrency-safe, and adds on_connection_create. Native async Spanner support is unchanged.

cofin added 30 commits October 31, 2026 21:31
Delete bool_to_int, resolve_column_names, detect_json_columns, and detect_json_columns_from_description from the aiomysql, asyncmy, mysqlconnector, and pymysql cores; the coercion maps register int directly. Drop the matching source-equivalence cases and the mysqlconnector create_mapped_exception exemption, since all four mappers are identical.
…ng transaction errors

The MySQL exception mapper returned True for duplicate-key-name and cannot-drop errors, so the driver handlers suppressed the exception without a result and statement execution failed with an AssertionError. Map both codes to OperationalError, make every handler record the mapped exception, and make begin/commit/rollback always raise the mapped error with its cause.
…ation

The Litestar session migration derives the store class from the config class name, so SqliteConfig, AdbcConfig, and DuckDBConfig failed to resolve SQLiteStore, ADBCStore, and DuckdbStore. Rename those stores, the DuckDB ADK classes, and the single-mode Sybase and SQL Anywhere event stores to match their configs, drop the old event store aliases, and add a guard test covering every adapter with a Litestar store.
… rows

The operation profile marked any DML with a returning clause as row-returning, including T-SQL OUTPUT ... INTO, which writes into a table variable. mssql_python now fetches rows whenever the cursor exposes a result set, so EXEC and DELETE ... OUTPUT results are not dropped.
…extension configs

FlaskConfig and StarletteConfig (and so FastAPIConfig) omitted enable_correlation_middleware, correlation_header, correlation_headers, and auto_trace_headers even though both extensions read them. Declare them and add a contract test that every key an extension reads is declared on its config type.
…ramework extension

Flask, Starlette, FastAPI, and Sanic defaulted enable_correlation_middleware to False while Litestar defaulted to True. All five now default to True; set the key to False to opt out.
…t null and check

Update SQLite and AioSQLite extended constraint error codes so NOT NULL uses code 1299 and CHECK uses code 275. This prevents trigger aborts from being falsely classified as not-null violations.
…ion, and error mapping

Prevent leaking dirty connections in connection pool release by retiring connections on rollback errors. Delegate cached query execution to cursor method to preserve ambient exceptions and format wire SQL, and map programming/interface errors and column names containing substring keywords accurately.
…ddings and parameterize event limit

Use WeakSet to track configured connections across ADK and Litestar stores to avoid ABA pragma skipping after address reuse. Implement binary float32 packing and similarity search for memory embeddings, and parameterize LIMIT in ADK get_events.
…adata caching

Hoist connection parameter filtering helpers and excluded keys to module level, eliminating per-call closures. Hoist cursor context in batch Arrow loading, cache version detection at class level in data dictionary, and memoize generated column DDL expressions per table.
…r mappings

Delegate script execution to base class with rowid cache invalidation, remove redundant pass-through load_from_storage, replace truthy or in pool creation with explicit is None checks, remove dead Python 3.11 version check, and add missing error mappings for auth, range, toobig, and notadb.
…ed memory cache, and error mappings

Roll back uncommitted transactions on exit in get_connection unless explicitly committed, track thread-local reentrancy depth, default memory database to cache=shared, delegate script execution to base driver with rowid cache invalidation, and map substring errors only when error_code is None.
…ze events limit

Implement binary float32 packing and vector similarity ranking for SQLite ADK memory store, parameterize LIMIT in get_events, and retain connection ID tracking for non-weakref-capable sqlite3.Connection instances.
…raise SQLFileNotFoundError

Memoize generated column expression parsing per table in SQLite and AioSQLite data dictionaries. Coalesce schema_name to 'main' in SQLite index and column introspection queries, and raise SQLFileNotFoundError when data dictionary queries are missing.
…int handling

Hoist connection parameter filtering helpers, epoch constants, and schema lists to module scope. Normalize cached query parameters, hoist cursor context in Arrow loading, and implement direct savepoint create/release without script splitting or rowid cache invalidation.
…ion lifecycle

Track active connection checkouts, prune dead thread connections on size inspection, default transaction mode to IMMEDIATE with ImproperConfigurationError validation, delegate commit and rollback to end_transaction, remove redundant load_from_storage, and remove unused logger.
…ache hit fast path

Ensure LOAD DATA statements escape percent characters in identifiers for PyMySQL pyformat interpolation. Check literal percent escaping in cached queries so clean queries delegate directly to the driver fast path without redundant statement allocations.
…L upsert syntax

Batch ADK memory entries with cursor.executemany rather than sequential execute calls in a loop. Drop redundant DROP INDEX statement from session table teardown and use VALUES(col) in Litestar session upsert statements for cross-version MySQL and MariaDB compatibility.
…te generator closures in compiled helpers

Add O(1) fast path to escape_literal_percent that returns the SQL string unchanged when parameter and percent counts match. Eliminate generator expressions from localized core helpers to avoid heap closures under MyPyC, and offload LOCAL INFILE tempfile creation and payload writes to asyncio.to_thread.
…te script execution, and clean up connection handling

Pass statement.statement_config to statement compilation and parameter validation across execution methods. Delegate dispatch_execute_script to the base driver implementation to emit statement observability events, close standalone connections on creation failures, catch AiomysqlPymysqlError consistently in transaction methods, and remove the redundant load_from_storage override.
Delegate _execute_cache_hit to super()._execute_cache_hit when cached SQL queries do not require literal percent escaping. This eliminates unnecessary statement and result allocations and restores direct cursor execution via collect_rows and resolve_rowcount.
… use cross-version upsert syntax

Batch ADK memory entries with cursor.executemany rather than sequential execute calls in a loop. Escape %, _, and \\ wildcards in memory search with an explicit ESCAPE clause, use LONGTEXT for content_text in the memory table DDL, remove redundant DROP INDEX from Litestar session teardown, and use VALUES(col) in Litestar session upsert statements for cross-version MySQL and MariaDB compatibility.
… add object_name filter to privileges

Combine VERSION(), @@sql_mode, and @@sql_quote_show_create scalar queries into a single SELECT statement and cache the resulting DDL context per driver session to minimize round-trips before SHOW CREATE. Add object_name filtering support to the privileges domain query in MySQL data dictionaries.
…d eliminate redundant dict copy

Offload LOCAL INFILE temporary file creation and payload writing to asyncio.to_thread in _load_from_arrow_via_local_infile, ensuring asynchronous cleanup with shutil.rmtree on both success and failure. Eliminate redundant dict copy of driver_features in AsyncmyConfig.__init__.
…mports, and streamline execution hooks

Add explicit MySQL errno mappings for 1054, 1146, 1040, 1290, and 1836 to handle asyncmy errors without SQLSTATE, remove Postgres 235xx codes, and close connection on SSCursor close error. Guard top-level asyncmy imports, relocate AsyncmySessionContext to config.py, set native Arrow/Parquet capabilities accurately to False, delegate script execution to base driver, and close standalone connections on creation failure.
…n causes in load_from_arrow, and map connection errors

Guard top-level mysql-connector imports using import_optional to prevent ModuleNotFoundError when optional dependencies are missing. Delegate stream fetching and load_from_arrow statement executions through _run_with_exception_handler and _check_pending_exception to maintain traceback chaining and exception context. Map PoolError to ConnectionTimeoutError and disconnected InterfaceError / errno 2055 to DatabaseConnectionError.
…andle missing tables, and align store DDL

Move conn.commit() inside the try block before rollback in sync append_event_and_update_state. Handle missing-table errors gracefully in sync and async ADK memory stores, remove redundant DROP INDEX from Litestar session teardown, use VALUES(col) in Litestar session upsert statements for cross-version MySQL and MariaDB compatibility, and align sync create_table lifecycle by removing redundant commit after execute_script.
…d shared metadata setup

Verify that data_dictionary properties on sync and async mysqlconnector drivers cache their instances. Use the consolidated scalar DDL metadata query and object_name filtering from the shared MySQL data dictionary.
…lizers, and compile column resolution

Precompute cursor_options in driver __init__ to avoid dictionary allocations on with_cursor calls. Cache _deserializer in stream source instances on initialization to eliminate per-chunk getattr lookups, and move _resolve_column_types and type code mappings into compiled core.py.
…, force unprepared DDL cursors, and clean config inheritance

Map MySQL errnos 1146 and 1054 to OperationalError instead of SQLParsingError, remove dead escape_literal_percent exports from core, and delegate script execution to the base driver. Force an unprepared cursor for TRUNCATE and LOAD DATA LOCAL INFILE in load_from_arrow to prevent errno 1295 when prepared statements are configured. Clean up connection context slots and remove redundant provide_connection and provide_session overrides in MysqlConnectorAsyncConfig.
cofin added 27 commits November 1, 2026 22:11
@cofin
cofin marked this pull request as ready for review November 2, 2026 03:14
@cofin
cofin merged commit b3b800d into main Nov 2, 2026
79 of 82 checks passed
@cofin
cofin deleted the adapter-remediation branch November 2, 2026 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants